A Rule-Based Question Answering System For Reading Comprehension Tests

We have developed a rule-based system, Quarc, that can reada short story and find the sentence in the story that best answers a given question. Quarc uses heuristic rules that look for lexical and semantic clues in the question and the story. We have tested Quarc on reading comprehension tests typically given to children in grades 3-6. Overall, Quarc found the correct sentence 40% of the time, which is encouraging given the simplicity of its rules. 1 I n t r o d u c t i o n In the United States, we evaluate the reading ability of children by giving them reading comprehension tests. These test typically consist of a short story followed by questions. Presumably, the tests are designed so that the reader must understand important aspects of the story to answer the questions correctly. For this reason, we believe that reading comprehension tests can be a valuable tool to assess the state of the art in natural language understanding. These tests are especially challenging because they can discuss virtually any topic. Consequently, broad-coverage natural language processing (NLP) techniques must be used. But the reading comprehension tests also require semantic understanding, which is difficult to achieve with broad-coverage techniques. We have developed a system called Quarc that "takes" reading comprehension tests. Given a story and a question, Quarc finds the sentence in the story that best answers the question. Quarc does not use deep language understanding or sophisticated techniques, yet it achieved 40% accuracy in our experiments. Quarc uses hand-crafted heuristic rules that look for lexical and semantic clues in the question and the story. In the next section, we describe the reading comprehension tests. In the following sections, we describe the rules used by Quarc and present experimental results. 2 R e a d i n g C o m p r e h e n s i o n T e s t s Figure 1 shows an example of a reading comprehension test from Remedia Publications. Each test is followed by five "WH" questions: WHO, WHAT, WHEN, WHERE, and WHY. 1 The answers to the questions typically refer to a string in the text, such as a name or description, which can range in length from a single noun phrase to an entire clause or sentence. The answers to WHEN and WHERE questions are also sometimes inferred from the dateline of the story. For example, (EGYPT, 1951) contains the answer to the WHEN question in Figure 1. Ideally, a natural language processing system would produce the exact answer to a question. Identifying the precise boundaries of the answer can be tricky, however. We will focus on the somewhat easier task of identifying the sentence that contains the answer to a question. 3 A R u l e b a s e d S y s t e m fo r Q u e s t i o n A n s w e r i n g Quarc (QUestion Answering for Reading Comprehension) is a rule-based system that uses lexical and semantic heuristics to look for evidence that a sentence contains the answer to a question. Each type of WH question looks for different types of answers, so Quarc uses a separate set of rules for each question type (WHO, WHAT,